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Abstract: This paper presents a case study where twelve graduate students 
undertook the demanding role of the adaptive e-course developer and worked 
collaboratively on an authentic and complex design task in the context of open 
and distance tertiary education. The students had to work in groups in order to 
conceptualise and design a learning scenario for adaptive learning, develop 
learning materials and adaptive learning strategies, implement the respective 
adaptive e-course and finally, reflect on their experience. The primary goal of 
this intervention was to engage the students in the whole lifecycle of a 
Technology Enhanced Learning design artefact. In teaching highly complex skills 
for the ill-structured domain of Learning Design the four components 
instructional model (4C/ID) was exploited. Students were exposed to a wide 
range of interconnected issues and made design choices. Mixed research 
methods are used to conclude on the effectiveness of the intervention. 
Implications of this study include design guidelines towards an environment that 
implements complex adaptive behaviour in today's learner-generated digital 
world where Computer Supported Collaborative Learning often converges with 
Computer Supported Collaborative Design. 

Keywords: complex learning, technology enhanced learning, adaptive learning, 
computer supported collaborative design 

Introduction 

In the field of Technology Enhanced Learning (TEL), the process of 
transformation from the initial conceptualisation of an ICT-infused learning 
design, its development using technological means and its enactment with 
learners plays a central role (Munoz-Cristobal, 2012). This transformation 
process, all important from the designer's standpoint, constitutes a demanding 
and complex task. It requires the mastery of an integrated set of competencies 
in order to effectively address issues that may range from learning theory to 
software or knowledge engineering (Mor & Winters, 2007; Sims & Koszalka, 
2008; Hardre & Kollmann, 2013; Paquette et al., 2006). The alignment of these 
competencies with a multidisciplinary approach towards learning design is 
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important also due to the lack of commonly- and cumulatively-built 
understandings within the field of Technology Enhanced Learning (TEL) 
community (Mor & Winters, 2007). In the "real world" that awaits the Learning 
Design practitioners after the completion of their studies, they will probably need 
to deal with "a range of interconnected issues including technological, 
organisational and pedagogical" (Conole, 2008). Among others, a learning 
designer in the field of TEL will be called to make decisions about: strategy, 
interactions, interfaces, content and its delivery (Sims, 2006),as well as, to 
accomplish tasks such as (Kenny et al, 2005): writing learning objectives, 
identifying the types of learning outcomes, selecting media formats, delivery 
modes etc 

The intervention described in this paper was enacted as part of a 14-week 
graduate course on "Learning Design and Design of Educational Software" which 
was included in an MSc in Information and Communication Systems (ICS) 
program during the first semester of the academic year 2012-2013. The course 
is offered online and it combines weekly synchronous sessions (using the web 
conference technique) with asynchronous activities implemented through the 
university learning management system and via email. The asynchronous 
activities mainly facilitated a) instructor-to-student and student-to-student 
communication and b) access to learning materials, assignments, assignment 
grades and so on. To a lesser extent, the learning management system 
facilitated the students' familiarisation and experimentation with certain learning 
technologies, like Web 2.0 technologies. 

Twelve students, aged from 25 to 45 years old, participated in the course. 
Students taking this course specialise in Educational Technology, usually for 
professional reasons. Six of them were active educators in different settings 
(elementary, primary, secondary and continuous/adult education), teaching 
diverse subject matters (informatics, mathematics, physics and others). Two of 
them were pre-service teachers and the remaining four had various non-teaching 
professions involving ICT. Four live in Cyprus and eight live in Greece. None of 
them had participated before in a lesson about LD or TEL. All of them had 
completed their studies in tertiary education. During the academic semester, 
students submitted ten short weekly and two in-depth assignments. 

The intervention discussed in this paper attempts to exploit the Learning Design 
approach in order to help novice instructional designers to: 

• adopt a more creative and holistic view towards the design of a TEL design 
artefact (Conole, 2010) 

• realise the interplay between the various dimensions and approaches that 
influence the Learning Design process and its enactment with learners 
using technological means 

• liaise effectively in a multidisciplinary team and communicate the needs of 
all the critical stakeholders of the field within the team 

Learning Design & IMS-Learning Design specification 

In the recent literature, the meaning of Learning Design is twofold (Conole, 
2008; Donald et al., 2009) since it can be viewed both: a) as a process of 
designing for learning (i.e. lessons, learning activities, lesson plans) and b) as a 
product that contains elements such as descriptions of learning tasks, resources 
and scaffolds. Dobozy (2011) expands this typology by adding the notion of 
Learning Design as "a standardised (re)presentation of technology enhanced 
learning sequences and prescribed design based procedures that are content 
independent". In (Conole, 2008) it is stated that the term in recent years is 
being used almost synonymously to "course design". For the scope of this paper 
LD is defined as "the act of devising new practices, plans of activity, resources 


2 of 20 



and tools aimed at achieving particular educational aims in a given situation" 
(Mor and Craft 2012). 

The term came into usage almost a decade ago with the development of the IMS 
Learning Design (IMS-LD) specification (IMS GLC, 2003a; Conole 2008). It aims 
to provide a means of formalisation of the teaching-learning process through an 
educational modelling language that employs the metaphor of a theatrical play 
(Koper & Tattersall, 2005). Using this metaphor, the IMS LD specification 
provides a pedagogical neutral representation of the teaching-learning process 
(Koper & Olivier, 2004) comprised of actors, roles and sequences of activities. 
From this perspective, the Learning Design specification is a notational system 
which IMS-LD compliant editors and players employ, along with instructional 
engineering methods (Paquette et al., 2006), in order to create a Learning 
Design for sharing and future re-use (Kordaki, 2007). 

• The information model of the IMS-LD (IMS GLC, 2003b) has three levels of 
implementation, each a superset of the precedent one: 

• level A provides the core elements of the modelling language (roles, 
activities, services, resources etc) 

• level B adds properties and conditions and serves as the adaptation basis 
since, in its simplest form, enables the creation of rule-based (simple 
"IF-THEN-ELSE" rules) adaptation: an event is triggered when a property 
satisfies a condition. For example, the average grade of the student 
("property") is calculated ("event") when the last question of a quiz is 
answered ("condition"). Adaptation rules are created by the designer 
during authoring time of the resulting Unit of Learning (i.e. the adaptive 
course) 

• level C adds notifications of events to users, as well as, notifications 
between system parts 

Adaptive educational systems are model-based systems that adapt some of their 
key functional characteristics - for example, content presentation and/or 
navigation support (Brusilovsky, 2003) - taking into account learner 

characteristics like learner needs and preferences (Magnisalis & Demetriadis, 
2012). The IMS-LD specification was frequently used as the modelling basis for 
adaptation and personalisation (Burgos et al., 2007; Paramythis & Loidl- 
Reisinger, 2003; Magnisalis & Demetriadis, 2012; Specht & Burgos, 2007) in a 
range of TEL examples that include: an adaptive learning management system 
(Boticario & Santos, 2007), authoring tools to define re-usable adaptive learning 
designs (Berlanga & Garcia, 2005; Miao, 2005; Sampson et al., 2005), adaptive 
learning designs (De-la-Fuente-Valentin et al., 2011; Berlanga & Garcia, 2006; 
Burgos et al., 2007; Mavroudi & Hadzilacos, 2012(a),(b)), Computer Supported 
Collaborative Learning (Hernandez-Leo et al., 2006; Valdivia et al., 2009; 
Magnisalis & Demetriadis, 2012). The adaptive learning strategies that can be 
fully or partially implemented with the use of the IMS-LD methodology can be 
summarised as follows (Burgos & Barak, 2009; Burgos et al, 2006): learning 
flow adaptation (fully supported), content-based adaptation (fully supported), 
interactive problem-solving support (fully supported), adaptive user grouping 
(partially supported), interface adaptation (partially supported), adaptive 
evaluation (partially supported) and adaptive information filtering and retrieval 
(not supported). 

Complex learning & 4C/ID model 

The design of the intervention is based on the Four Components Instructional 
Design model (4C/ID), a model originally developed by Van Merrienboer et al. 
(2002), who suggested that environments for complex learning can have four 
interrelated components: 
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a. Learning tasks that engage students in activities suitable for the 
development of the needed constituent skills, as opposed to activities 
where students need to study general information related to the skills. 

b. Supportive information that bridges what students already know to their 
work on the learning tasks. Tutors typically refer to this type of information 
as "the theory", often presented in lectures and textbooks. 

c. Just-In-Time (JIT) information relates to the constituent skills that should 
be performed similarly in different problem situations. It offers students 
information about the procedural knowledge they need to obtain in order to 
perform the recurrent skills. Examples of this type of information include 
instructions provided during students' practice, where tutors are acting as 
an "assistant looking over your shoulder" (Van Merrienboer et al., 2002). 

d. Part-task practice which is required if a very high level of automaticity of 
particular recurrent aspects is necessary. Examples of part-task practice are 
"drilling children on multiplication tables and playing scales on musical 
instruments" (Van Merrienboer et al., 2002). An example of part-task 
practice in training design of an air traffic controller might involve critical 
recurrent constituent skills in terms of safety, for example, identifying risky 
air traffic situations from a radar screen (Van Merrienboer et al., 2002). 

Two main principles of the model are: 1) scaffolding and fading (i.e. withdrawing 
help as the learner progresses) and 2) that in each learning task, the complexity 
of the sub-tasks should gradually increase. Finally, the model aims at the 
automation of the recurrent aspects of the task, while promoting deep learning 
for its non-recurrent aspects. According to Van Merrienboer et al. (2003) 
complex learning involves "the integration of knowledge, skills and attitudes, the 
coordination of qualitatively different constituent skills and the transfer of what 
is learned to daily life or work settings". According to Merrill (2002), the model is 
based on problem-based learning and also focuses on a learning activity that 
employs complex cognitive skills. 

The basic aim of our intervention was to make the students appreciate the 
interdisciplinary nature of the field of educational technology and the interplay of 
the various perspectives and dimensions involved in the design of a TEL artefact 
through a problem-based learning approach. On top of that, the use of 
IMS-Learning Design specification provided a formalisation of the teaching¬ 
learning process through the metaphor of the theatrical play, leaving all the 
design decisions up to the learning designer. In parallel, the focus was also 
placed on Learning Design systems that, in the context of this paper, are defined 
as "workflow engines for collaborative activities" (Udas, 2009). Two IMS-LD 
compliant systems were used by the students in order to author and preview 
their adaptive e-courses. The resulting learning environment was an Open 
Learning Environment (OLE) with tools, resources and activities suitable for the 
promotion of divergent thinking in a learning situation where multiple 
perspectives are valued (Hannafin et al, 1983). Learning Design is an 
open-ended and ill-defined problem. Thus, the learning tasks were designed so 
as to provide enough opportunity to the students for practice both the non 
recurrent (study LD theory, design learning scenario, employ an adaptive 
learning strategy and so on) and the recurrent aspects of the complex skill (drill 
the most commonly used functionalities of the tools and so on). That is why such 
a learning ecology was appropriate and why the 4C/ID model was a suitable 
model for our students' collaborative tasks. 

More specifically, our students had to deal with the following tasks: 

1. read the respective LD theory, which, for the scope of this intervention, 
comprised two chapters from a previous master thesis on the topic, the 
IMS-LD Information model (IMS GLC, 2003b), the IMS-LD Best Practice 
and Implementation Guide (IMS GLC, 2003c), one chapter from the text 
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book (Koper & Tattersall, 2005) and lesson slides 

2. study worked-out examples of adaptive learning scenarios and their 
incarnations in the form of Units of Learning; worked-out examples and 
their corresponding UoLs were administered to the learners (two examples 
of level A and two examples of level B and their corresponding UoLs) 

3. conceptualise and design a learning scenario for adaptive learning close to 
their interests (i.e they could select any subject matter); towards this end, 
a learning scenario template accompanied with the description of the 
semantics of all its fields was administered to the learners 

4. draw a UML activity diagram to demonstrate the different phases, actors, 
interactions, synchronisation points between actors and learning strategies 
used in the learning scenario-see examples in (IMS GLC, 2003c) 

5. develop learning materials and adaptive learning strategies that can be 
implemented fully or partially with the use of the IMS-LD methodology 

6. install and use the authoring tool (i.e. the MS-LD compliant editor), as well 
as, a tool to preview their work (i.e an IMS-LD compliant player); user 
guides were provided 

7. implement the respective adaptive Unit of Learning (i.e. adaptive 
e-learning lesson) using the authoring tool and frequently preview the 
results using the player 

8. discuss the difference concerning the implementation difficulty between a 
non-adaptive (Level A-compliant) UoL and an adaptive (Level-B compliant) 
UoL 

9. reflect on the concepts of "re-usability", "learning content", "learning 
services", "learning activities" and the use of widgets (see Annex 2) 

10. reflect on their experience by proposing a model of guidance and support 
in an imaginary scenario where they had to lead a multidisciplinary team in 
order to construct adaptive Units of Learning -this task was included as a 
topic in the students' final exams 

11. map textual descriptions of a set of events (what constitutes an event was 
described in the previous section) against the type(s) of adaptation 
functionality they implement (e.g. learning flow adaptation, content-based 
adaptation, interactive problem-solving support, adaptive user grouping 
etc), this task was also included as a topic in the students' final exams 

Methodology 

The contextual settings of the Learning Design intervention 

The intervention took place between the 10 th and the 13 th week of the course 
(from the time that the assignment was announced to the students to the time 
that the students' products were uploaded to the Moodle LMS). The two last 
sub-tasks mentioned in the previous section were included as sub-topics in the 

j_i_ 

final exams, due one week after the end of the 14 week of the course. The 
participating roles in this intervention were: a) the tutor, who was the main 
person responsible for the course in general and was basically responsible for the 
support of the students with the more "theoretical aspects", b) the assistant, 
who was the main person responsible for the more practical parts i.e. of the 
design of the worked-out examples and completed scenarios and c) the 
students. In terms of scaffolds, except those mentioned above, one web 
conference meeting (where the students, their tutor and the assistant 
participated) was devoted to the enlightenment of the difficult parts of the 
theory. Additionally, support and advice was provided by the assistant via a 
dedicated online forum, Skype and email. It revolved mostly around tool 
functionality issues, but secondarily also involved other issues like ideas about 
adaptation parameters. Towards this end, the paper by Economides (2008) was 
also included in the suggested readings. The support included synchronous as 
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well as asynchronous discussions. Student evaluation rubrics and performance 
standards were designed collaboratively by the tutor and the assistant (the 
students' evaluation rubric is shown in Annex 1). 

The twelve students who participated in the course were asked to work 
collaboratively and consequently, three groups were formed. In each group there 
was at least one member who was an educator (and, as such, he/she was 
familiarised with lesson planning procedures) and at least one member that was 
an informatics professional. 

In terms of the IMS-Learning Design compliant tools used, the Recourse LD 
editor and the Astro LD player were pre-selected by the tutors (both available 
at: http://tencompetence-project.bolton.ac.uk/ldauthor/) 

Data collection methodology 

Empirical results were gathered from the following data collection sources: 

1. the web-conference recording session, 

2. posts in the dedicated forum in the university LMS, 

3. answers on an online survey questionnaire and follow-up semi-structured 
interviews 

4. the final products of the students i.e. their designed artefacts 

5. the answers of the students in the hypothetical scenario (a final exams 
question) 

6. the log-files of the LMS system and the IMS-LD player (the online version) 
Some remarks concerning the data collection methodology: 

• The online survey questionnaire was distributed to the students 
immediately after the end of the semester and the semi-structure 
interviews followed a few weeks afterwards. The interviews duration was 
20 minutes in average and they were conducted through electronic means 
(VoIP and the Elluminate live! ™ web-conference software, according to 
the student's preference). 

• The log files provide some evidence about the type and the frequency of 
students' interactions with the system, but it is not a clear and reliable 
indicator concerning their engagement and participation. This is due to the 
fact that, concerning the LMS forum, it was only used for the "official story" 
of the students' interactions, since from the interviews it became evident 
that the students used Skype™ to collaborate on their assignment. As far 
as the IMS-LD complaint player is concerned, each student had access both 
to the online version but also to their local, offline version of the player 
that they used to preview and test their UoLs. 

• Participation in the online survey, in the discussion forums or in the 
personal interviews was voluntary. Also, none of the above contributed 
towards the students' final grades. The return rate of the online survey was 
100%, since all the students completed the survey. The return rate of the 
semi-structured interviews was 67%, since eight students participated (the 
remaining four students were either not able or willing to participate). 
Finally, the students were aware of the fact that the web conference 
sessions and the semi-structured interviews were recorded. 

Data analysis and results 

Collaboration 

The students felt that the aspect of genuine collaboration was addressed 
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satisfactorily (n = 12, mean=4.6, standard deviation=0.6 in a five-point Likert 
scale where the respective question in the online survey was "This task favoured 
genuine collaboration". A score of 'one' indicated failure in satisfying the goal 
and a score of 'five' indicated success). 

In the online survey questionnaire the students were asked to reason about the 
score that they had assigned in the previous question. The open-ended question 
was used in the survey in order to provide the students with the opportunity to 
make suggestions, comments and complaints related to their learning experience 
(Kanuka & Anderson, 2007). The transcripts of the open-ended questions were 
analysed using the grounded theory approach (Glaser & Strauss, 1967; Strauss 
& Corbin, 1994), a content analysis method that allows key themes to be 
emerged mostly from textual data through coding cycles. The code, concepts 
and categories emerged were tested against the interview transcripts. The 
grounded theory was selected as opposed to an interpretivist approach (e.g. 
theory-driven approach) of qualitative data analysis, since the research goal was 
not to test any preconceived hypothesis about the students' collaboration and 
engagement but rather to try to understand their ontological meanings in a 
contextual way (Allan, 2003). The results are shown in the textbox, where 
concepts are presented in the form of bulleted lists and the emerged themes are 
in capital letters. 

Figure 1. Concepts and themes concerning the collaboration aspect 


TEAM MEMBERS COLLABORATED IN ORDER TO RESOLVE PROBLEMATIC 
SITUATIONS 

• Collaboration as a means of resolving tool issues and providing help 

• Collaboration in order to compensate of lack of supportive material 

ACTIVE TEAM MEMBERS HAVE A POSITIVE ATTITUDE TOWARDS 

COLLABORATION 

• Positive attitude/comments towards collaboration 

• Collaboration perceived in terms of equal contribution by team members 
COLLABORATION WAS FOSTERED THROUGH A SCHEDULE 

• Having a schedule (task schedule, meeting schedule) 

• Collaboration via frequent meetings between team members 

• Conflictions with other students' obligations hinder participation 

COLLABORATION WAS SUPPORTED BY A RANGE OF CSCL TOOL AND 
PRACTICES 

• Collaboration via online meetings 

• Collaboration perceived in terms of efficient communication 

• CSCL mediated via students' designed artefacts (lesson scenario, UoLs) 

• Tools and practices covered various aspects of CSCL (skype meetings for 
synchronous communication, forum for asynchronous communication, 
google docs for theoretical parts, desktop sharing for technical or practical 
issues) 

COLLABORATION SCHEMATA 

• Limited collaboration between teams 

• Role/task distribution within teams 

• Frequent controls and corrections 
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Concerning the collaboration via the forum, five discussion forums (each 
addressing a separate aspect of the students' assignment) were set up on the 
Moodle LMS. Two of them ("forum 1" and "forum 2") were created and initiated 
by the tutor and the remaining three by one particular student ("forum 3", 
"forum 4" and "forum 5"). The data presented in the table below indicate the 
frequency of student and instructor participation, the total number of messages 
in each thematic forum (Hara et al, 2000) and the number of students that 
participated in each forum discussion. 

Table 1. Participation in the online forums 


Name 
(created & 
initiated) 

Total number 
of messages 

Total number 
of instructors' 
messages 

Number of 

students' 

participants 

Forum 1 

(by the instructor) 

20 

9 

4 

Forum 2 

(by the instructor) 

9 

3 

4 

Forum 3 

(by the student) 

4 

1 

2 

Forum 4 

(by the student) 

29 

8 

7 

Forum 5 

(by the student) 

11 

10 

1 


The dimensions of content analysis using CSCL tools, such as an online forum, 
that are commonly employed are: participation, cognitive processing and social 
interactions (Sing and Khine, 2006). The data of the table above gives a rough 
idea of the participation aspect. The five discussions included altogether 73 
postings. The proportion of postings by the instructors was 42% (31 messages). 
This relatively large proportion can be attributed to the fact that the forum was 
perceived by the students as the means of getting help and support primarily 
from the professor and the assistant. In the context of this distance education 
setting, communication with them exclusively took place online. From the online 
survey and the semi-structured interviews it became evident that the students 
frequently consulted the forum. This is confirmed by the web analytics that 
enable the user tracking functionality of the LMS, which revealed 763 views of 
forum discussions (in the forums 1-5 and during the time period that the 
intervention took place). 

Concerning the types of postings, various content analysis schemes to analyse 
transcripts of online asynchronous discussions exist (Hara et al, 2000; De Wever 
et al, 2006). For the scope of this paper, the theoretical framework of Jarvela 
and Hakkinen (2002) seemed most appropriate, since it proposes a 
categorisation of the postings which discriminates between: (a) theory, (b) new 
point or question, (c) experience, (d) suggestion, and (e) comments. The 
message served as the unit of analysis for this categorisation. The students' 
messages in the forum were analysed and coded independently by the assistant 
and the author of the master thesis included in the students' readings. The 
inter-rater agreement percentage was at first 81%, which reached 95% after 
discussions about the disagreements between the two coders. The final results 
are depicted in the figure below. Out of the 42 students' messages posted in the 
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forum, 1 message (2%) concerned LD theory, 9 messages (21%) introduced a 
question or a new point, 10 messages (24%) suggested a solution, 10 messages 
(24%) reflected students' learning experience and 12 messages (29%) mostly 
commented on various other aspects related to the intervention at stake. 

Figure 2. Distribution of posting types 



The forum served as a common place of reference that included a significant 
number of suggestions concerning the above issues. It should be noted however 
that communication between the assistant tutor and the students was also to a 
lesser extent enacted through email and Skype. From the online survey it can be 
concluded that, overall, students felt that the support they received was very 
satisfactory (n = 12, mean=4.4, standard deviation^ 0.8 in a five-point Likert 
scale where the respective question in the online survey was "The support I 
received via the forum/Skype/email was satisfactory". A score of 'one' indicated 
failure in satisfying the goal and a score of 'five' indicated success). 

Bridging the gap between LD theory and IMS-LD practice 

The degree in which the intervention accomplished the goal of bridging the gap 
between theory and practice was perceived by the students as very satisfying 
(n = 12, mean=4.4, standard deviation= 0.9 in a five-point Likert scale where the 
respective question in the online survey was "The task helped me in bridging the 
gap between theory and practice". A score of 'one' indicated failure in satisfying 
the goal and a score of 'five' indicated success). 

Two indicators were evaluated positively on this criterion: 

1. Consistency between the learning scenario, the UML workflow diagram and 
its incarnation in the form of a Unit of Learning (see Annex 1). 

2. Evidence that the students could distinguish the difficulties that arose due 
to the user-friendliness of the tools they used (or the absence of it) from 
those that would remain the same regardless of the IMS-LD compliant tool 
being used. The latter implies difficulties inherited in the complexity of the 
conceptual framework of the IMS-LD (the workflow, the theatre metaphor, 
the adaptation elements). 

Concerning the first indicator, all the groups managed to achieve consistency 
between the learning scenario and the UoL, whereas none of the groups 
managed full consistency between the UML activity diagram and the learning 
scenario (or the UoL), although the students were familiarized with UML activity 
diagrams before their enrolment in the course. As far as the second indicator is 
concerned, all the students that were interviewed showed explicit evidence that 
they could distinguish the IMS-LD specification from the tools that implement it. 
An example of what constituted explicit evidence concerning the issue is 
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depicted below: 


(Student) If you have understood the IM-LD specification and you know that you 
have to cope with activities, roles, environments etc, and furthermore, if you can 
distinguish levels A, B & C, then -using the Recourse editor or any other IMS-LD 
compliant tool -you won't be having any issues, from the specification 
perspective. If you have conducted a good planning on how to implement it (i.e 
the learning design) then it's a purely technical matter and it depends on how 
familiarised you are with the program. 

In addition to the indicators mentioned above, the authors also detected some 
student statements that on the one hand exemplified deep understanding 
concerning the fact that the UoL is the incarnation of a learning design, but, on 
the other hand, they did not fall into either of the two categories mentioned 
above. An example is the following: 

(Student) After this assignment everything was clear to me, I didn't need to 
study (for the final exams), only some lectures. The only thing I took with me in 
the exams (students were allowed to bring with them any learning materials 
they wished for the final exams) was the printed form of the manifest.xml of the 
UoL we had created. The entire course is included in this file, what else did I 
want? [..] declarations, metadata, properties, activities, roles-parts, roles.., 
everything is in there. If you can conceive this, this XML file can tell you all you 
need to know. 

Additionally, from the discussions held with the students it became evident that 
they could abstract their learning experiences. By the time that the interview 
took place one student (an informatics professional) had already managed to 
transfer the knowledge gained from the intervention into her workplace. In 
particular, she mentioned that she had applied the design philosophy behind the 
IMS-LD compliant tool into a new e-business system in her work, in order to 
better conceptualise its information model since she felt that it was similar to the 
IMS-LD Information Model (i.e. resembling the metaphor of a theatrical play). 
Another student, who used two different IMS-LD compliant editors, could 
effectively communicate to the assistant during the semi-structured interview 
their different affordances and intended uses. 

Learning progress 

As mentioned in section "Complex learning & 4C/ID model", two sub-topics 
about adaptive learning were included in the students' final exams, aimed at 
checking how the students' learning about the topic at stake (i.e. adaptive 
learning designs) had progressed over time. The score of each team (i.e. the 
average score of the team members) both for their assignments and for the final 
exams was calculated. The results are depicted in the table below. 

Table 2. Scores & learning progress 


Team 

Assignment 
scores 
(out of 10) 

(Average) exam 

scores 

(out of 10) 

Team A 

8.5 

7.3 

Team B 

9 

9.3 

Team C 

9.5 

9.7 
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As it is shown in the table, there is a strong positive correlation (r=.92, p = .25) 
between the scores in the assignment with the scores (in the topics about 
adaptive learning) in the final exams. The correlation between the students' 
scores in the assignment and the students' scores in the final exams is depicted 
in the table below. The t-test still reveals a positive correlation between the two 
variables (r=.501, p = .01). Note that Kolmogorov-Smirnov tests were 

performed to test the normality presupposition. 

Table 3. Correlation between performance in the assignment 
and performance in the exams 
at the level of the individual student 


Correlations 



assignments 

exams 

Pearson Correlation 

1 

.501 

assignments Sig. (2-tailed) 


.097 

N 

12 

12 

Pearson Correlation 

.501 

1 

exams Sig. (2-tailed) 

.097 


N 

12 

12 


With regards to the hypothetical scenario where the students had to lead a 
multidisciplinary team in order to construct adaptive Units of Learning, most of 
them reproduced their own variations of the 4C/ID model. 

Complex learning in today's digital world: CSCL or 
CSCD? 

Background and rationale 

There is a growing body of research on co-design for learning purposes 
approached from two major perspectives: 

• In terms of specific-purpose tools; for example in (Hernandez-Leo et al, 
2011) a web-based authoring environment, "LdShake", is presented as a 
tool that enables the co-edition and social network-oriented sharing of 
learning designs created using a general rich text editor. 

• In terms of collaborative design of ICT-infused learning scenarios by 
teachers, as a form of their professional development. This approach sees 
teachers as designers (Voogt et al, 2011) and is aligned with the 
well-established "learning by design" notion (Kolodner et al., 2003), while 
it acknowledges that the active involvement of the teachers in the LD 
process might have a positive impact on their professional development 
and in turn, on student learning (Kali & McKenney, 2012). 

In parallel, the question of how we could apply to LD insights derived from the 
discourse with other design disciplines is an interesting aspect that has begun to 
flourish and to attract the attention of the stakeholders (Mor & Craft, 2012; Mor 
et al., 2013). Examples of initiatives that embrace this aspect include: 

• The Learning Design studio (LDS), an effective manifestation of the Design 
Inquiry of Learning (DIL) model. The latter combines an inquiry-based 
learning approach with a design-based scientific paradigm. The former is 
modelled after the tradition of studio-instruction in arts and design 
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disciplines, such as architecture, its main characteristic being the students' 
on-going group work on design challenges in a domain of practice (Mor & 
Mogilevsky, 2013). 

• The Design Principles Database (DPD), which aims at synthesizing 
emerging design knowledge about the use of technologies for education 
(Kali,2006). 

• The International Journal of Designs for Learning, a journal dedicated to 
publishing descriptions of artefacts, environments and experiences created 
to promote and support learning in all contexts by designers in any field. 

In this study the design of the collaborative activities and, consequently, the 
evaluation of students' designed artefacts were focused on two challenges that 
students needed to face through group work, e.g.: 

• the formulation of an adaptive learning strategy which poses several 
challenges for the novice designers, since it involves the interplay between 
the adaptation parameters (prior student knowledge, learning objectives, 
student performance in learning tasks etc.) with the adaptation methods 
(like: sequencing of learning activities, feedback, student grouping etc.). 

• The interplay between the different LD representations in the lifecycle of 
the students' design artefacts. The learning experience involved the design 
of a learning scenario and the development of the corresponding unit of 
e-learning. Literally speaking, the Unit of Learning is the incarnation of 
their learning scenario. 

Creation of artefacts and collaboration were the two most pivotal design 
patterns, while the following design principles were incorporated in the 
intervention: 

• use open-ended construction tools 

• engage learners in a complex project 

• use multiple representations 

• provide knowledge representation and organisation tools 

• promote autonomous lifelong learning 

• provide students with templates to help reasoning 

• provide just-in-time data to students 

• connect to personally relevant contexts 

• encourage reflection 

Implications 

The purpose of this section is to provide insights concerning the interactions 
between the students with their digital artefacts, as well as their social 
interactions in order 1) to discuss educationally effective strategies in a complex 
learning task where Computer Supported Collaborative Learning (CSCL) and 
Computer Supported Collaborative Design (CSCD) took place and 2) provide 
future directions on how they can be incorporated in the design of an IMS-LD 
environment. 

In particular, focus is placed on the role of the mediating artefacts viewed from a 
distributed cognition framework perspective, since the latter "is specifically 
tailored to understanding interactions among people and technology" (Hollan et 
al, 2000) in terms of not only what people know, but how they go about using 
what they know to do what they do (Hollan et al, 2000). For the scope of this 
paper (where the participants' observation was not possible) distributed 
cognition is being viewed from a situative perspective (rather from a cognitive 
one) in which knowledge exists in the way that social groups communicate, 
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make use of symbols, tools and designed artefacts and understanding is a 
process of negotiating the meaning of these objects with others (Hewitt and 
Scardamalia, 1998). Interesting points in the intervention at stake are: 1) the 
use of student-created designed artefacts and 2) the students' spontaneous use 
of CSCL tools and techniques, in order to mediate their learning and reduce their 
cognitive load. The discussion is mostly based on what is presented in the 
section titled "Data analysis and results", sub-section titled "Collaboration". As 
already mentioned in that section, CSCL emerged mostly within teams, whereas 
among teams it has commonly happened in the forums. Also, with regards to the 
collaboration schemata: roles and subsequent tasks distribution among team 
members: additionally frequent controls and corrections took place. CSCL was 
perceived by the team members in terms of efficient communication (i.e. how 
well ideas and experiences were communicated among team members) and 
equal contribution to the tasks. The students' constructed mediated artefacts 
were: the lesson scenario and the UoL. As already mentioned, the students used 
a variety of CSCL tools and practices: Skype meetings for synchronous 
communication, forum for asynchronous communication, Google docs for 
theoretical parts (i.e. related to the creation of the learning scenario), desktop 
sharing for technical or practical issues (related to the creation of the UoL) in a 
spontaneous mode. Concerning the organisational and social settings that 
superseded the learning process, students acted as lifelong learners, since they 
took full responsibility of their progress by preparing a shared plan of action 
(task schedule, meeting schedule) and organising frequent online meetings. As it 
was expected, participation was hindered by students' obligations (family 
obligations, work obligations). On the other hand, having a positive attitude 
towards collaboration and being an active member within a team seem to be 
interrelated. Finally, collaboration between students was practiced as a means of 
resolving tool issues, providing help and compensating the lack of supportive 
material. 

In conclusion, we argue that the incorporation of a CSCL tool in a way that 
fosters a variety of practices in today's digital world where learners create and 
share their artefacts could be an indispensable characteristic of an 
all-encompassing IMS-LD compliant environment. More specifically, concerning 
the desktop sharing feature in the case of CSCD of the UoL, it will enable the 
"what you see is what I see" (WYSIWIS) design principle which is suggested in 
previous work for designing at a distance via real-time designer-to-designer 
interaction (Scrivener et al, 1993). The benefits of Web 2.0 tools (such as forum, 
chat, Google docs etc.) have been extensively discussed in the CSCL-related 
literature in general and especially with regards to professional development 
(Cochrane & Narayan, 2013). Finally, a shared view of the designed artefacts 
along with the use of Web 2.0 tools would foster the communication and the 
support coming from the experts, something that seems to be vital in the case 
of novice developers. 

Discussion 

This paper is a design case where CSCL and CSCD took place, in which it is 
evident that the students were not pre-occupied with the Learning Design 
aspects that are closely related to pedagogy. On the one hand, it has been 
argued that students who follow Learning Design courses offered via computer- 
science related programs of study (in this case, the ICS program) face difficulties 
in the design of learner-centred courses which revolve around the design of 
appropriate lesson plans (Kordaki et al, 2007). On the other hand, the creation 
of learner-centred e-courses was not the focus of this particular intervention, 
although exemplifying sound pedagogical choices was considered an advantage 
(as one can see in Annex 1). 

With regards to our methodology, the idea of conducting semi-structured 
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interviews with the possible stakeholders for eliciting design requirements is not 
new, since they can provide direct access to 'experience' (Silverman, 2000). For 
example, Luck (2000) suggests requirements about the inclusive design of a 
building based on interviewing people with disabilities. Where there were 
conflicting requirements, these were resolved during dedicated steering group 
meetings. Yet, a building is a physical entity and as such it can have only one 
facade, whereas this is not true for a digital environment intended for 
collaborative design of Units of Learning. Since multiple views in such an 
environment would be possible, future plans include further work in the 
requirements analysis by identifying possible different requirements for the two 
main groups: teachers and informatics professionals. Yet, no conflicting 
requirements were noted in our case. 

Finally, the benefits of adaptive learning to students' cognitive development 
have long been discussed (Lee & Park, 2007; Kim, 2012; Hwang et at., 2012), 
thus excluding the teachers from the creation of rich adaptive learning designs 
may hinder their professional development. 
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Annex 1: The students' evaluation rubric 

A. Answer in the question of Annex 2 (15 points) 

Clear and concise answers concerning the concepts: re-usability, learning 
material, learning services, learning activities, widgets 

B. Unit of Learning (40 points) 

1.1. Technical integrity/excellence (15 points) 

1.2. In accordance with the learning scenario (25 points) 

C. Learning scenario (30 points) 

1.3. Are all the fields satisfactorily completed/addressed? (10 points) 

1.4. Do the students address thoroughly the topic of each field? Is it evident 
that they have understood the semantics of the field? (10 points) 

1.5. Is the workflow diagram correct? Is it fully consistent with the learning 
scenario? (10 points) 

D. Overall impression (articulated answers, authenticity, within word limits, 
pedagogical rationale and roles, complexity of adaptive strategies: does it 
combine more than one adaptive strategy? If yes, how many? The combination 
concerns the same or different phases of the scenario?) (15 points) 

Annex 2: A question that was included in the students' assessment 

Upload and run the completed (ready-made) UoLs to the LD player in order to 
preview the respective e-course and to formulate an initial idea about their 
design and enactment, as well as, the degree and the nature of their 
re-usability. In order to answer to the latter think that the reusability might 
entail processes and justifications like the following: 

- remove this learning content, keep the lesson structure intact and insert my 
own content 

- remove this learning content, change the lesson structure by adding or 
removing phases or learning activities and also insert my own learning content, 

- .... etc 

Please describe (in no more than 150 words) your learning experience. Having 
practiced the above, what do you think about the concepts of "learning content", 
"learning services", "learning activities"? 

By now, you must have read two chapters from the master thesis. The process 
of learning design described in a chapter of the master thesis integrates some 
widgets. Have you understood how they can be used? 
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